Internal State GPOMDP with Trace Filtering

GPOMDP is an algorithm for estimating the gradient of the average reward for arbitrary Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. It applies to purely reactive (memoryless) policies, or policies that generate actions as a function of finite histories of observations. Based on the fact that maintenance of a belief state is sufficient for optimal control in POMDPs, this paper extends GPOMDP to cover parameterized stochastic controllers with internal state. We also generalize the discounting of rewards used by GPOMDP and other RL algorithms to arbitrary IIR and FIR filters, and show how prior knowledge may be used to set the taps in the filters in a way that reduces variance in the gradient estimation. Several experimental results are presented, including large scale phoneme recognition.